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Tracking Student Achievement in Science and 
Math: The Promise of State Assessment Programs 



The National Science Foundation is investing millions 
of dollars in efforts to improve math, science, and 
technology education through its Statewide Systemic 
Initiatives Program. However, states are facing diffi- 
cult challenges in accurately assessing the effects of 
these efforts on student learning. 

Launched by NSF in 1990. the SSI program aims to 
address widespread concerns about student perfor- 
mance in .science and mathematics in the United 
States, and related concerns about inequities in student 
performance associated with race, ethnicity, and 
gender. NSF solicited proposals from states willing to 
create broad-based coalitions to undertake compre- 
hensive and ctxirdinated reforms of science, math, and 
technology education to address these problems. NSF 
made 10 awards in 1991. 11 in 1992 and 5 in 1993 .' 
The typical award was for as much as $10 million 
over a period of five years. The states have provided 
substantial matching fhnds to support their initiatives. 
The resulting state initiatives are often referred to 
collectively as the SSls. 

Naturally, given the magnitude of this investment in 
education reform, state and federal policymakers are 
anxious to know whether the SSls are having the 
desired effects on student performance in mathematics 
and science. Are students learning more because of 
these state initiatives? Are they better equipped to 
apply what they are learning to everyday problems in 
the community and in tlie workplace? Are the inequi- 
ties in performance among different groups of 
students being reduced? Answering these questions 
requires good methods of assessment. 



This issue of CPRE Policy Briefs examines the 
capacity of state assessment systems to track the 
effects of the SSls on student performance in mathe- 
matics and science. It also identifies some of the 
major issues state policymakers are facing as they 
attempt to re-align their state assessment systems to 
meet the changing goals for public education.’ 

The Importance of Appropriate 
Assessment 

The SSls are founded on ideas about good practice in 
mathematics and science education that are widely 
shared among education reformers and have gained 
considerable currency among teachers. These ideas 
about what constitutes good curriculum and in- 
struction and what students should know and be able 
to do in the two disciplines have been promoted and 
disseminated by national and state professional 
groups. A “national vision” of good practice in 
mathematics seems to be emerging from the Na- 
tional Council of Teachers of Mathematics’ Stan- 
dards, and in science from the American Association 
for the Advancement of Science's Benchmarks for 
Scientific Literacy, the National Science Teachers’ 
Assix:iatioiis' The Content Core: A Guide for Curricu- 
lum Designers, the National Research Council’s Draft 
National Science Education Standards and related 
state dcKuments.’ 

As described in these sources, good practice is 
characterized by challenging academic standards; a 
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hands-on approach to instruction: 
active learning by students; the 
use of curricula and materials rel- 
evant to students’ lives; an em- 
phasis on thinking, problem- 
solving. and the application of 
knowledge; the integration of 
mathematics, science, and the use 
of technology in the classroom; 
and the use of assessment that is 
consistent with the more rigorous 
content and reinforces these 
instructional practices. 

For example, in Montana, where 
the SSI has concentrated on de- 
veloping and implementing new 
high schtxil curricula, an observer 
saw the following mathematics 
lesson demonstrating the kind of 
instruction reformers are trying to 
promote:^ 

Michael is leaching stuilents about the 
mathematics of exponential growth, 
using populations of animals as an 
e.xample...lSi'tudents working in small 
groups are asked to make up rules for 
the life ami death of two populations of 
animals, represented by two t\jes of 
candies shaken in a ho.\... A rule 
relating to proximity of candies might 
he a model for predation -for example 
if a large and small candy are less than 
an inch apart, take the smaller one out 
of the box (it dies) and put an 
additional larger one into the box. 

Using the set of rules they create, they 
set the populations going for about A 
life cycles, or A shakes of the box. The 
student groups then move to one of the 
eight computers in the room, recording 
their data in a computer spreadsheet, 
and producing a graph of the two 
populations over time, w hich thev print 
out. Students r;ust he able to produce 
mathematical expressions using 
e.xponeniial notation that match the 
grow th or decline of these populauons. 

Assessing the effecls of this kind 
of classroom practice on students' 
knowledge of mathematics and 
science requires appropriate as- 
sessment instruments. Conven 
tional multiple-choice tests pro- 
bably cannot adequately capture 
what students are learning in this 
Montana classromn. What are 
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needed are assessment procedures 
that examine students’ ability to 
explain important concepts, apply 
knowledge to new problems, and 
integrate information across 
fields. Appropriate assessment 
procedures might include essays 
or open-response questions, com- 
plex multiple-choice questions, 
performance tasks completed by 
individuals or groups of students, 
or portfolios of student work. 
Assessments such as these could 
generate the kind of student 
performance information needed 
to answer fundamental questions 
about the effectiveness of the 
changes i.i curriculum and class- 
room practice being encouraged 
by the SSIs. 



The Policy Studies 
Associates Survey 

Will state assessment programs 
provide the information about 
student performance needed to 
answer policymakers' questions 
about the effectiveness of the 
r>Sls? To answer this question. 
NSF asked its national evaluation 
team in the spring of 1994 to 
examine the capacity of the SSI 
states to assess student perfor- 
mance in mathematics and science 
and provide goixJ information 
about the effects of the SSIs.' One 
of the cooperating organizations. 
Policy Studies .Associates (PSA) 
undertook a survey of the current 
assessment capacity in science and 
mathematics in the SSI states. 

In the spring and summer of 
1994. PSA interviewed state as- 
sessment staff and SSI staff in the 
25 states.'’ Based on data collected 
through these interviews, the PSA 
study team examined the potential 
of existing state assessment 
systems to produce the data 
needed to track the effects of SSI 
efforts on student performance. 
Specifically, it asked whether 
achievement in mathematics and 
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science was assessed statewide, 
and whether testing data were 
available at the appropriate grade 
levels, were reported for the 
appropriate unit of analysis, and 
were consistent for a sufficient 
period of time to be useful in 
evaluating the SSI. It also ex- 
amined whether the states had 
assessment systems that were 
aligned with the learning goals 
embodied in the SSIs. Their find- 
ings. based on the status of state 
testing programs during the 1993- 
94 scluxM year, are outlined below. 

Assessment in the SSI 
States 

What is the character of the 
existing state assessment prv 'grams 
in mathematics and science? 
Tables 1 and 2 summarize PSA's 
findings about these programs in 
the SSI states for the 1993-94 
school year. The most obvious 
conclusion to be drawn from these 
data is that more states assess 
students in mathematics than in 
science. As Table 1 show's, all but 
3 of the 25 states assessed students 
in mathematics. The three excep- 
tions were Nebraska. Montana, 
and Colorado which had no 
statewide assessment systems at 
all. In contrast Table 2 shows that 
only 16 SSI states collected any 
data on studer' achievement in 
science. In addition to Nebraska. 
Montana and Colorado. 6 other 
states had no statewide science 
assessment. 

Student performance data for 
sueiice is less available primarily 
because science has not been 
considered a “basic" subject. 
State assessment programs had 
their origins in the basic skills 
movement of the 1970s and typ- 
ically included reading, writing, 
and basic mathematics, but not 
science. While state assessment 
programs have raised their stan- 
dards and added more challenging 
content in recent years, many still 
do not include science. 



Table 1; Required Statewide Assessments as of 1993-94: Mathematics 





Years 
of SSI 




Grades 


Item 




Reporting 


State 


Funding 


Assessment Program Name 


Tested 


Type 


Scoring 


Levels 


Arkdnsas 


93-98 


Stanford Achievement Test-8 (SAT-8) 


4,7,10 


MC 


N 


D.S.I 


Arkansas Minimum Performance Test 


3,6,8 


MC 


C 


D.S.I 


California 


92-97 


California Learning Assessment System 


4,8,10 


MC,P 


PL 


D,S 


Colorado 


93-98 


No Statewide Assessment 


— 


— 


— 


— 


Connecticut 


91-96 


Connecticut Mast- y Test 


4,6,8 


MC.P 


C 


D.S.C.l 


Connecticut Acadt 'lic Performance Test 


10 


MC.P 


c 


D,S,1 


Delaware 


91-% 


Delaware Interim Assessment 


3,5,8,10 


MC.P 


PL 


D.S.I 


Florida 


91-% 


Grade Ten Assessment Test 


10 


MC 


N 


D.S.I 


High School Competency Test 


11 


MC 


C 


D.S.I 


Georgia 


92-97 


Iowa Test of Basic Skills/TAP 


3,5,8,11 


MC 


N 


D.S.I* 


Curriculum-Based Assessment 


3,5,8,11 


MC 


C 


D,S 


Kentucky 


92-97 


Kentucky Instructional Results Information 
System 


4,8,12 


P.PF 


PL 


D.S.C 

D.S.I 


Louisiana 


91-% 


Louisiana Educational Assessment Program 


3,5,7,10 


MC 


C 


California Achievement Tcst-5 


4,6 


MC 


N 


D.S.I 


Maine 


92-97 


Maine Educational Assessment 


4,8,11 


MC.P 


N 


D.S.I 


Massachusetts 


92-97 


Massachusetts Educational Assessment Program 


4,8,10 


MC.P 


PL,N 


D,S 


Michigan 


92-97 


Michigan Educational Assessment Program 


4.7,10 


MC 


C 


D.S.C, I 


Montana 


91-% 


No Statewide Assessment 


— 


— 


— 


— 


Nebraska 


91-% 


No Statewide Assessment 


— 


— 


— 


— 


New Jersey 


93-98 


Early Warning Test 
High School FYoficiency Test 


8 

11 


MC.P 

MC.P 


PL 

C 


D.S.I 

D.S.I 


New Mexico 


92-97 


Iowa Test of Basic Skills* 


3,5,8 


MC 


N 


D 


High School Competency Exam 


10 


MC 


C 


D 


New York 


93-98 


Pupil Evaluation Program 


4 


MC.P 


C 


D,S 


Regents Compentency Tests** 


9 


MC.P 


c 


D.S.I 


North 


91-% 


End-of-Grade Tests 


3-8 


MC.P 


N 


D,S 


Carolina 


End-of-Course Tests 


9-12 


MC.P 


N 


D,S 


Ohio 


91-% 


State Proficiency Test 


9,12 


MC 


C 


D 


Puerto Rico 


92-97 


APRENDA (Spanish version of SAT) 


1-9 


MC 


N 


R.S.I 


South Carolina 


93-98 


Basic Skills Assessment Program 


3,6,8,10 


MC 


C 


D.S.C.l 


Stanford Achievement Test-8 


4,5,7,9,11 


MC 


N 


D.S.C.l 


South Dakota 


91-96 


Stanford Achievement Test-8 


4,8.11 


MC 


N 


D 


Texas 


92-97 


Texas Assessment of Academic Skills (TAAS) 


3,8,10 


MC 


C 


D.S.I 


TAAS— End-of Course 


9-12 


MC 


C 


D.S.I 




92-97 


Portfolios 


4.8 


PF 


PL 


D 


Vermont 


Uniform Assessment 


4,8 


MC.P 


PL 


D 


Virginia 


92-97 


Iowa Test of Basic Skills 


4,8,11 


MC 


N 


D.S 


Literacy Passport Test 


6-8 


MC 


C 


D 1 



Key: Item Type: MC = Multiple Choice, P = Performance and Open-ended Items, PF = Porttolios, Projects, and Experiments 

Scoring: N = Norm-referenced, r = Criterion referenced, PL = Performance Levels 

Reporting l,cvels: R = Region, D = District, S = School, C = Classroom, I = Individual 



♦At the 1 1th grade, Georgia uses the Test of Achievement and Proficiency (TAP). Students taking the I AP are matrix sampled 
so that scores cannot be reported at the individual student level. Item type and scoring remain the same. 



**To receive a diploma, students must demonstrate competency in mathemalhics by passing either the Regents Competency Test 
or a Regents examination in mathematics. The two assessments listed in this table are those likely to Ix; most relevant for assessing 
the SSI. 
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Table 2: Required Statewide Assessments as of 1993-94: Science 





Years of 














SSI 




Grades 


Item 




Reporting 


State 


Funding 


Assessment Program Name 


Tested 


Type Scoring 


Levels 


Arkansas 


93-98 


Stanford Achievement Test-8 (SAT-8) 


4,7,10 


MC 


N 


D,S,I 


Arkansas Minimum Performance Test 


3,6,8 


MC 


C 


D,S,I 


California 


92-97 


California Learning Assessment System 


5, (8.10) 


MC,P 


PL 


D.S 


Colorado 


93-98 


No Statewide Assessment 


— 


— 


— 


— 


Connecticut 


91-96 


Connecticut Academic Performance l est 


10 


MC.P 


C 


D.S.l 


Delaware 


91-96 


No Statewide Assessment 


— 


— 


— 


— 


Florida 


91-96 


No Statewide Assessment 


— 


— 


— 


— 


Georgia 


92-97 


Curriculum-Based Assessment 


3,5,8,11 


MC 


c 


D,S 


Test of Achievement and Proficiency (TAP) 


11 


MC 


N 


D,S 


Kentucky 


92-97 


Kentucky Instructional Results Information 
System 


4.8.12 


P,PF 


PL 


D.S.C 


Louisiana 


91-96 


Louisiana Educationa Assessment Program 
Califo;:nia Achievement Test-5 


3,5,7,10 

4,6 


MC 

MC 


C 

N 


D.S.I 

D.S.l 


Maine 


92-97 


Maim Educational Assessment 


4.8.11 


MC.P 


N 


D.S.I 


Massachusetts 


92-97 


Massachusetts Educational Assessment Program 


4.8,10 


MC.P 


PL.N 


D,S 


Michigan 


92-97 


Michigan Educational Assessment Program 


4,7.10 


MC 


C 


D.S, CM 


Montana 


91-96 


No Statewide Assessment 


— 


— 


— 


— 


Nebraska 


91-96 


No Statew ide Assessment 


— 


— 


~ 


— 


New Jersey 


93-98 


No Statewide Assessment 


— 


— 


— 


— 


New Mexico 


92-97 


Iowa Test of Basic Skills 


3,5,8 


MC 


N 


D 


High School Competency Exam 


10 


MC 


C 


D 


New Vork 


93-98 


Pupil Evaluation Program 


3,6 


MC 


c 


D.S.I 


Regents Compentency Tests* ** 


9 


MC,P 


c 


D.S.I 


North 


91-96 


End-of-Grade Tests 


3,6,8 


MC 


N 


D.S 


Carolina 


End-of-Course Tests 


9-12 


MC 


N 


D.S 


Ohio 


91-96 


No Statewide A.ssessment 


— 


— 


— 


— 


Puerto Kico 


92-97 


No Statewide Assessment 


— 


— 


~ 


— 


South 

Carolina 


93-98 


Basic Skills Assessment Program 


3,6,8 


MC 


C 


D,S,C,I 


South Dakota 


91-96 


Stanford Achievement Tcst-8 


4.8,11 


MC 


N 


D 


Texas 


92-97 


TAAS— End-of Course 


9-12 


MC 


C 


D.S.I 


Vermont 


92-97 


No Statewide Assessment 


— 


— 


— 


— 


Virginia 


92-97 


Iowa Test of Basic Skills 


- 


MC 


N 


D.S 



Key: Item Type: MC = Mulliplc Choice. P = Performance and Open-ended Items, PF- Portfolios, Projects, and Experiments 

Scoring: N = Norm-referenced. C =Criterion-rcfcrcnced. PL = Performance Levels 

Reporting Levels: R = Region. D-= District. S-School. C -('lassrooni. {^Individual 



*The science portion of the ITBS is not required as part of New Mexico’s statewide assessment program; however, stale assessment 
staff report the 99 percent of the .-iiale’s districts administer the lest. 

**To receive a diploma, students must demonstrate competency in science by passing either the Regents C'ompelency Test or a 
Regents examination in science. Ilie two assessments listed in this table arc those likely to be most relevant for assessing the SSL 



Taking Stock of the SSIs 

Tills is the second i.ssue of CPRE Policy Briefs to 
examine the National Science Foundation's 
Statewide Systemic Initiatives Program. The first, 
O I “Reforming Science, Mathematics, and Technology 



F:ducation: NSF’s State Systemic Initiatives." 
provides an overview of the SSI program. It also 
describes the policy strategies and implementation 
problems being faced by the 25 state SSIs. 



Limitations of State 
Assessment Data 

Knowing that student assessment 
results in science and mathematics 
are available is not sufficient to 
determine their usefulness for 
evaluating the SSIs. The student 
performance data must also be 
consistent with the emerging 
standards in mathematics and 
science and fit the goals of the 
SSI. To determine whether the 
results of state assessment pro- 
grams can be used to assess state 
initiated reforms in math and 
science education, policymakers 
need to address such questions as: 

• Do the state tests assess rigorous 
content, problem-solving, and 
applications of knowledge in the 
disciplines? 

• How well-aligned is the content 
of the state tests with the cur- 
riculum and instruction promoted 
by the SSI? 

• Are students tested at the grade 
levels targeted by the SSI? 

• Does the state report scores at 
the appropriate level of the educa- 
tional system (e.g.. if the reform 
effort concentrates on schools, 
does the state report student 
achievement data at the school 
level)? 

• Does the state administer tests 
in mathematics and/or science that 
yield comparable data statewide'.’ 

• Will the state's assessment pro- 
gram remain consistent for long 
enough to allow for its use in 
evaluating effects of reform ef 
forts on student achievement over 
time? 

The Character of State 
Assessment. I’.SA found that 
despite the considerable national 
fanfare about the introduction of 
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portfolios and performance-based 
assessments and the raising of 
academic standards, state testing 
programs were still relying heavi- 
ly on multiple-choice formats and 
were focused on basic skills. As 
the tables show, the assessments 
included a mix of item types. 
Tests composed solely of 
multiple-choice items were the 
most common. In mathematics. 12 
of the 25 SSI states were using 
multiple-choice tests exclusively, 
while 9 others had at least one test 
that combined multiple-choice 
items with open-ended questions 
or performance events. In science. 
10 states were using tests that 
were solely multiple-choice while 
5 were using tests with mixed 
formats. Only 2 states. Vermont 
and Kentucky, were relying 
primarily on performance-based 
assessments, and Vermont was 
doing so only in mathematics. 
Clearly, state assessment pro- 
grams had not yet abandoned 
more traditional forms of testing. 

Yet. fundamental changes in state 
issessment policies and practices 
i.ave taken place in states such as 
California. Connecticut. Kentucky 
and Vermont. In these states, 
performance-based assessment 
systems have been instituted as 
part of their broader reform 
strategies. However, with the 
possible exception of Connecticut 
where new forms of assessment 
have been introduced incremen- 
tally. these new approaches to 
assessment are surrounded by 
controversy. Public concern about 
the character of tlie new assess 
me lit system in California 
produced political turmoil that led 
to its suspension, and. probabls 
will result in its replacement witii 
a more conventional assessment 
system. And critical studies have 
raised questions about the techtii 
cal adequacy of the new system in 
Kentucky atid generated public 
debate about the future of the 
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state's assessment program. (The 
sidebars on pages 6. 7. and 8 high- 
light the cutting-edge assessment 
systems being used in Connecti- 
cut. Kentucky, and Vermont.) 

Meanwhile, other states like 
Arkansas. Delaware, and Massa- 
chusetts are developing new 
performance-assessment systems. 
In Arkansas, a 1991 education 
reform act eliminated the state's 
Minimum Performance Test and 
replaced it with a 12th-grade exit 
exam to be aligned with the state’s 
new curriculum frameworks and 
learner outcomes. In addition, the 
state is instituting benchmark 
exams at grades 4 and 8 and is 
piloting a portfolio assessment. 
Delaware adopted an Interim 
Assessment Program in 1992-93 
to be used until work on curricu- 
lum frameworks and a new 
performance-based system could 
be completed. The Interim As- 
sessment. obtained from a com- 
mercial testing company, includes 
open-response items. In Massa- 
chusetts. the 1993 educational 
reform act called for phasing out 
the state's testing program and 
developing curriculum frame- 
works and coordinated assess- 
ments. including a “competency 
determination" as a requirement 
for high school graduation. 

.Some states are revising existing 
assessment programs. Michigan, 
which recently upgraded its 
mathematics assessment to make it 
consistent with the NC I'M stan- 
dards. is also upgrading its sci- 
ence assessment for 1995. I.ouisi 
ana plans to revise its existing 
state assessments in mathematics 
and science after new ciiiriculuin 
frameworks are completed. North 
Carolina is revising and e.xpand 
ing its end-of-grade assessmenis 
in inatheinatics and science. 

Another grcuip of states including 
l lorida. New .lersey. .South Caro- 
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Connecticut’s State Assessment Program 

Connecticut, a slate with a strong tradition of local control, has used 
its state assessment program to stimulate changes in local curricula. 
Connecticut has a two-pronged approach to assessment: the Connec- 
ticut Mastery Test (CMT) and the Connecticut Academic Performance 
Test (CAPT), both developed by the state. The CMT is a criterion- 
referenced test in mathematics and language arts, given every fall to 
students in grades 4, 6, and 8. First administered in 1985, the CMT 
was revised for 1994-95 to incorporate more problem-solving and 
higher-order skills, primarily by using more demanding and realistic 
open-ended questions. 

Implemented in 1994, the CAPT assesses mathematics, science an ’ 
language arts and is given every spring to students in grade P'. flie 
CAPT includes multiple-choice and perfoniiance items. As p:.rt of the 
science assessment, for example, students are given i' hands-on 
laboratory activity before the administration of the test id then are 
asked follow-up questions that relate directly to the hanus-on task. 

The CAPT also includes an interdisciplinary assessment section, in 
which students read and analyze a set of source materials that ftKus on 
“a controversial topic of social significance." Students then write a 
{lersuasive piece, using the source materials, to support their position 
on the issue. CAPT scores are included in students' trails- ''ipts. 

The results of these state assessments are reported at the individual, 
classrcxmi, schcxil, district, and state levels. They also are included in 
school profiles that are published annually by the Department of 
Education and receive broad media coverage. 



lina, and Virginia appear to be 
content to rely on their basic skills 
tests. A few states like Montana 
and Nebraska may never create 
statewide assessment systems be- 
cause of strong traditions of local 
control and public concern about 
content and costs. 

In addition to differences in the 
types of assessment items used, 
state tests also differ sharply in 
their scoring methods. Norm- 
referenced tests are constructed to 
compare achievement across large 
groups of students and are not 
linked to specific school or district 
curricula. Criterion-referencecl 
tests, on the other hand, measure 
student mastery of specific 
curriculum objectives and provid. 
valid performance comparison!, 
only across schools and districts 
using those objectives, riiey also 
can be used to assess the quality 
and effectiveness of a curri- 



culum's implementation. Both 
types of tests are used widely in 
state assessment programs. 

(yf the 25 slates studied. 12 were 
using at least one mathematics 
testing program that was norm- 
referenced; 13 were using at least 
one testing program that was 
criterion-referenced. In science, 
the ratio was approximately the 
same. Few states report scores in 
terms of student performance 
levels, a method of scoring 
associated with newer, perfor- 
mance-based tests which require 
students to apply knowledge by 
solving a problem or conducting 
an experiment. Six states report at 
least some student results in terms 
of performance levels in inallie- 
nia ICS, and only three do so in 
scii nee, 

Content Alignment. Ihe PSA 

study found that the degree of 



content alignment between state 
assessments and SSI goals for 
curriculum and instruction ranged 
widely. At the high end of align- 
ment are those SSls which are 
integral parts of broader reform 
efforts. In Kentucky, for example, 
many individuals participating in 
the state's SSI also play active 
roles on state curriculum and 
assessment committees. SSI staff 
see the state's new assessment 
system as closely aligned with the 
curriculum and instruction they 
wish to promote and have de- 
signed the SSI to support its im- 
plemeniaiion. 

At the other end of the scale, 
there is weak alignment, and 
changes in state assessment pro- 
grams are lagging far behind other 
reform efforts. For example, 
while it is clear that state tests 
designed to measure minimum 
competency cannot capture out- 
comes in mathematics and science 
that the SSls are attempting to 
promote, slates such as Florida, 
New Jersey, New Mexico, and 
Virginia continue to administer 
them. In Virginia, for example, 
tin SSI's definition of math and 
science achievement includes 
d.mionstrated student under- 
standing of conceptual connec- 
tions in mathematics and science; 
inquiry, problem-solving and de- 
cision-making; and the value of 
mathematics and science. How- 
ever. Virginia's primary state 
assessment instrument, the Iowa 
Test of Basic Skills, cannot mea- 
sures the SSI's success in promot- 
ing tliese outcomes because it 
focuses mainly on achievement in 
basic skills. 

Similarly Puerto Rico's multiple- 
choice. iHinn-relerenccxl .APRFiND.A 
testing program does not measure 
the higher-order thinking skills 
the SSI is promoting in its middle 
and elementary schools. The com- 
monwealth's SSI has overcome 



Kentucky’s State Assessment Program 

Adopted in 1990, the Kentucky Instructional Results Information 
System (KIRIS), is designed to provide data for both school 
accountability and instructional improvement. KIRIS is a high-stakes, 
performance-based assessment system keyed to four student 
proficiency levels; t is aligned with a new state curriculum framework 
based on the state’s six learning goals and 57 academic expectations. 
The KIRIS assessment has three parts: 

1 . Open-response items covering reading, math, social studies, 
science, arts and humanities, and practical living/vocational studies are 
included in the assessment. Open-ended questions in each subject tap 
multiple areas of content knowledge and require several steps to an- 
swer. The test allocates approximately 15 minutes for each open-ended 
response. 

2. Students must also complete a one-hour performance event in math, 
science, social studies, arts and humanities, or practical living/voca- 
tional studies. The students work on an experiment or problem-solving 
activitN in small groups or pairs and write their answers individually. 

3. Students also submit portfolios containing five to seven examples of 
their best work in both writing and mathematics. Mathematics 
portfolios are submitted in grades 5. 8. and 12; writing portfolios are 
submitted in grades 4, 8 and 12. High school portfolios are under 
development. Scores from the mathematics portfolios, assessed 
statewide for the first time in 1992-93. will be incorporated into the 
accountability index starting 1994-95. Writing portfolio scores account 
for 14 percent of the cognitive part of the accountability index. 

Various components of this assessment system are administered in 
grades 4. 5. 8. 11. and 12. Results are reported as the percentages of 
students attaining the four proficiency levels. KIRIS results are 
combined with other indicators including attendance and graduation 
rates u> produce a score for each school on an accountability index. 
Kach school is given a target, based on its previous baseline, for 
impriiving its score on the index over a two-year period. Schools 
receive rewards on assistance depending on their progress, 



this deficiency by developing its 
own assessments. New Jersey's 
Early Warning Test and High 
School Proficiency Test also 
assess minimum competency, not 
the kinds of higher-order, perfor- 
mance-based learning outcomes 
h>r which the state’s SSI strives. 
Conversely, in these states tlie 
SSIs may push for reforms in 
curriculum, pedagogy, and class- 
room assessment that may do little 
to ensure that student scores 
increa.se on more tr;iditional state 
tests, 

Grade-Level Coverage. Most 
SSIs are K 12 initiatives. 1 low- 
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ever, of the 22 states with state- 
wide mathematics assessments. 4 
lack adequate coverage to evaluate 
K-12 state systemic initiatives; 
that is. they do not test students in 
one or more of the elementary-, 
middle- and high-school levels. 
Ohio and florida provide state- 
wide testing only at the high- 
school level, Vermont currently 
assesses students only in grades 4 
and S. and New Jersey adminis- 
ters a statewide test only in grades 
X and 1 1.' Of the 16 states with 
statewide science assessments. 4 
fail to cover all three levels, 
Connecticut and Texas test 
students only at the high-school 



level. South Carolina only in 
grades 3, 6, and 8, and New York 
only in grades 4 and 9. 

Reporting Levels. To be useful 
in evaluating an SSI, the analysis 
of the results of the statewide 
assessment must match the SSI’s 
intervention strategy. If the stra- 
tegy targets individual schools, an 
assessment program that reports 
student-achievement data only 
down to the district level— such as 
those in Ohio, New Mexico, and 
Vermont— will not help to isolate 
the impact of reform efforts. 
Reform efforts that target indi- 
vidual teachers also present spe- 
cial problems. In Arkansas, for 
example, the state reports student- 
achievement data at the district, 
school and individual student 
levels; but it may be impossible to 
sort scores by classrooms in order 
to evaluate effects of training on 
individual teachers. The task 
becomes even more difficult when 
teachers do not have self-con- 
tained classrooms, a common ar- 
rangement at the secondary - 
school level. 

These assessment problems can be 
overcome. The Louisiana Depart 
ment of E.ducation has worked 
with the SSI to modify its testing 
forms so the stale can link 
individual student scores w'ith 
teachers who have received SSI 
training. But remedies require 
deliberate planning and add 
additional costs. 

A related problem is that while 
appropriate data may exist, they 
may be difficult to ohtiiin. .School- 
level data are hard to obtain in 
such states as Ohio and New 
Mexico, where each district must 
decide whether to release tlie 
information to the public. In 
Ohio, the process of obtaining 
permission to examine school- 
level data from local superin- 
tendents has hindered the progress 
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Vermont’s State Assessment System 

Vermont’s assessment system relies on portfolios in mathematics and 
writing and the Uniform Assessment. The system aims to encourage 
effective instructional practices and provide information for public 
accountability and program evaluation. 

In 1990, the state education agency began piloting portfolios in writing 
and math in grades 4 and 8. The portfolio system was implemented 
statewide in 1991-92. All schools and districts now participate in the 
voluntary program. The state collects and scores samples of mathe- 
matics portfolios in grades 4 and 8, writing portfolios in grades 5 and 
8. The state board of education will implement mathematics portfolios 
in the high schools in 1995-96 after the end of a three-year pilot pro- 
gram. 

Each portfolio is unique, containing five to seven best pieces ot work 
selected by the students from their regular class work. Students are 
expected to revise the entries and may work on them as long as they 
choose. The scoring rubric used to assess the math portfolios encour- 
ages the submission of the complex, multi-step problem-solving activi - 
ties advocated in the NCTM standards. 

The scoring of the portfolio sample collected by the state is done by 
teachers who have been trained within regional networks established 
by the state. The central scoring of the state sample produces results 
by district. 

The regional networks train teachers in portfolio management and 
scoring. Local districts also hold conferences to score all ot their 
portfolios. About UK) schools plan to score portfolios from the 1994- 
95 school year and will report their results in the fall. Eventually, 
school-lt*' el scores will be reported tor all schools. 

The Uniform Assessment covers mathematics and writing. It is 
designed to supplement the portfolio process with an on-deniand, 
standardi 7 .ed assessment. The mathematics section includes complex 
multiple-chiiice items taken from the National Assessment ot 
Educational Progress and one open-ended problem. Vermont expects 
to replace the Uniform Assessment with a more performance-based 
assessment in the near future and plans to add a standardized science 
assessment in 1995-96. 



of the state SSI evaluation. As a 
consequence, the Ohio SSI is 
adininist' ng its own perfor- 
mance assessment in the class- 
rooms ot teachers \.ho attend its 
intensive suninier institutes in 
mathematics and science. 

Consistency and Stability of 
the Measures. Twenty -four of 
the 25 states surveyed required 
some kind of standardized testing 
of students at specific grade levels 
or grade-level clusters in 1993- 
94. The one state that did i.ot. 



Colorado, plans to institute new' 
tesiMig requirements in 1996-97. 
Of the 24 SSI states currently 
requiring some form of testing, at 
least 6 (1-1. . MT. NE.. N.I. Oil. 
and SD) allow some local choice 
about the tests to be administered 
at some or all of the tested grade 
levels. The PSA study did not 
attempt to determine how useful 
these “local-choice" testing 
programs would be for evaluation 
of systemic reform, but they do 
raise a problem of equating results 
on different tests. 



Constantly changing state assess- 
ment programs furthtn compound 
the difficulties of evaluating the 
impact of the SSIs on student 
performance. Changes in the 
nature or timing of the assess- 
ments make it difficult to deter- 
mine changes in student perfor- 
mance over time. For example, 
Massachusetts is in the midst of a 
transition to an entirely new 
assessment program. Changes are 
also underway in California. In 
New York, the futures of the 
Pupil Evaluation Program and the 
Regents Competency tests are 
unclear. Overall, 14 of the 22 
states with mathematics assess- 
ments and 12 of the 16 states with 
science assessments plan to 
change or expand at least one of 
their tests during the next few 
years. Such changes make it 
difficult to follow trends in stu- 
dent achievement. 

Comparability of Assess- 
ments Across States. The 

variety of state SSI objectives and 
the diversity of state assessment 
programs limit the use of student- 
assessment data for comparisons 
of SSI programs across states. 
Comparisons can be made across 
the small number of states that use 
the same commercially developed 
tests, but these tests, although still 
in use in many local districts, are 
being de-emphasizeo as states 
change their statewide assessment 
programs. In 1993-94. among the 
25 SSI states, only 8 administered 
norm-referenced tests purchased 
from national publishers; 4 used 
the Stanford Aci.ievement Test 
statewide, 3 used the Imva Test of 
Hiisic Skills and 1 used the Cali- 
fornia Achievement Test. At least 
5 other states required districts to 
administer a commercially pub 
lished test chosen from a list of 
state-approved assessments. All 
told, about half of the SSI states 
were using these tests. 



Summing up: State 
Capacity for Evaluating 
the SSis 

The PSA study found only six 
state assessment programs in 
mathematics and four in science 
that met all of its criteria for 
evaluating the SSIs. That is, 
assessment programs which test at 
the appropriate grade levels, 
report data down to the appro- 
priate level of the educational 
system, are expected to provide 
data over five years of the life of 
the SSI, and have test content and 
formats aligned with the cur- 
riculum and instruction promoted 
by the SSI. 

It is clear that there are only a few 
states in which available student- 
assessment data are adequate for 
an evaluation of systemic reform 
in math and science education. In 
others, reformers will have to 
work with serious data limitations 
until their state assessment sys- 
tems can be aligned more closely 
with the goals of systemic reform 
efforts. In most cases, the SSis 
would have to administer addi- 
tional student assessment in order 
to provide a complete picture of 
their effects on student perfor- 
mance in mathematics and sci- 
ence. Only a handful of SSis are 
doing so. The potential for using 
student-assessment data collected 
by the states to e\ aluate the SSI as 
a federal strategy is limned. 

Issues Facing State 
Policymakers 

The PSA study suggests tliat the 
data needs of policymakers exceed 
the capacity of existing state 
assessment programs to measure 
progress toward reform goals. 
This is particularly true for sci 
ence. Only ib of the 25 SSI sttites 
currently have any statewide 
assessment in science. And onlv a 
O 
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few of those appear to be aligned 
with the emerging science 
standards. Standards-driven re- 
form requires new assessments 
that are better aligned with the 
changes being sought in curri- 
culum and pedagogy. While con- 
siderable work is underway to 
develop such assessments, formid- 
able barriers must be surmounted 
to develop and implement valid 
and reliable assessments that are 
consistent with the emerging 
standards m mathematics and 
science. The development of ade- 
quate state assessment programs 
requires addressing some thorny 
issues such as tho.se discussed 
below. 

Purposes of Assessment 

States art seeking new' assessment 
systems that can simultaneously 
serve multiple purposes: pro- 
viding good data for comparing 
the performance of ;,chools, 
providing diagnostic feedback to 
help teachers improve their 
practice and meet the needs of 
individual students, and providing 
strong incentives for desired 
changes in practice. This seems 
reasonable on its face, but it may 
be difficult in practice. Assess- 
ments measuring individual stu- 
dent achievement and providing 
instructional feedback may not 
necessarily be good tools for 
accountability. And it may be 
hard to keep the multiple purposes 
in balance. Thus a high-stakes 
assessment may drive curriculum 
and pedagogy in unanticipated, 
and undcsired, directions, while 
resulting in short-term improve 
ments in performance uiulcrmin 
mg long-term goals for iinprox ing 
practice. 

Validity of Methods 

Arc the new alternative assess 
ments more valid than multiple 
choice tests as their proponents 
claim.’ That is. do they more 
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accurately measure the domains of 
knowledge or attributes of 
individuals that the reforms in 
classroom practice are intended to 
emphasize? Do they provide 
information that is consistent with 
other measures of the same 
domain? Are they goexi predictors 
of the future performtmee in 
school oi at work? 

In the case of performance 
assessment and portfolios, the an- 
swers to these and other questions 
about validity are not yet clear. 
Reliability (the consistency of the 
ineasui cment from one time to the 
next) is also an issue for some of 
the new assessments. Providing 
reliable scores for individual stu- 
dents on performance assessments 
may take more time and be more 
costly than conventional multiple- 
choice tests. Portfolios raise spe- 
cial concerns about reliability. 
I-fforts to "improve" portfolios 
in order to increase the reliability 
of the scores are viewed by some 
teachers as undermining their 
value as instructional tools and 
levers for classroom reform. 

Consequences 

Closely related to the issue of 
purpose is the question of the 
appropriate stakes to be assiKiated 
with assessment. If the assessment 
system is linked to accountability, 
who should be held accountable, 
what should the consequences be. 
and who should bear them'.’ 
Should there be rewards as well as 
sanctions? Should money be used 
as a rewaril'.’ Shoukl school take 
overs or closings be considered as 
ultimate sanctions? What kinds of 
data arc needed to justify the 
distribution of rewards aiul sane 
tions? These issues are among the 
most eiiK'tional and hotly debated 
(|Ucs(ions about asst sment poliev . 

High-stakes tests are viewed by 
some as narrowing the curriculum 
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and negatively affecting both 
students and teachers. Con- 
versely, others argue that low- 
stakes tests offer little incentive 
for teachers to make the difficult 
changes required in curiiculum 
and pedagogy or for students to 
work harder and select more 
difficult courses of study. The 
question of who should bear the 
consequences is also debated. 
Some see schools and their staffs 
as bearing primary responsibility 
for student performance. Others 
argue that students and schools 
sliould share responsibility and the 
consequences. In most states, 
there are stakes for students 
(graduatu)ii, promotion, school 
records) and for teachers and 
schools (public reporting, pro- 
gram changes, state intervention 
or takeover!. How to define stakes 
so tliat the press for reform is 
strong but not distorting is the 
issue. 

Breadth vs. Depth 

There is some lension between the 
reform slogans "high standards 
for all." "less is more." and 
"hands-on learning." and the use 
of statewide standardized assess- 
ments. I he slogans suggest that 
all children slmuld meet a uniform 
set of academic standards and that 
the standards should be consistent 
with a constructivist pedagogy 
that focuses on understanding of 
key concepts and applications. 
This approach trades off breadth 
for depth of understanding, and 
idciilly would leave teachers aiul 
schools some discretion ahout 
these curricular decisions so that 
the areas of depth c;ui be rehited 
to students' backgrounds, in- 
terests. and prior learning. This 
poses serious challenges toi the 
design of state assessment sys- 
tems. If the assessment covers too 
brotid a domain ot knowledge, 
teachers may experience difficulty 
helping students prepare for it. On 




the other hand, researchers have 
found that when there are high 
stakes associated with a test, the 
provision of specific guidance 
about what will be tested often 
narrows the curriculum to what is 
covered by the test and eliminates 
local discretion. There appears to 
be no simple resolution to this 
dilemma, but balance must be 
sought between instructional re- 
forms promoting depth at the ex- 
pense of breadth of curriculum 
coverage and the assessment 
system. 

Professional 

Development 

The emerging assessment systems 
make substantial demands on 
teachers. Teachers not only need 
to understand the requirements of 
the new systems, but in many 
instances, they are expected to 
change their practice, enhance 
their subject-matter knowledge, 
develop new curricula, and serve 
as overseers and assessors in the 
new' prtK’ess. They also are on the 
front line in dealing with patents 
who may direct their concerns and 
misgivings about the tiew as- 
sesstiients to teachers (especially 
where teachers do the scoring as 
with portfolios). They need op 
portunities to acquire the neces- 
sary knowledge and skills, to 
practice new strategies, and to 
interact with other teachers about 
what w'orks and how' to solve 
common problems. In short, a ra- 
dically restructured and refocused 
system of professional develop- 
ment is needed. The system must 
be intensive, continuous, aiul 
connected to classroom practice. 

Development Time 

Technically adequate pert', |•mancc 
assessments will take time to 
develop and perfect. Vermont has 
beeti working on portfolio assess- 
metU in tnath for five yc s and 



otily now is on the verge of 
meeting conventional psychome- 
tric standards. And it has only 
begun to implement assessmetits 
for the high school. Kentucky's 
new assessment system is in its 
fourth year and has undergone 
critical review by two external 
groups. 

These time fratnes for develop- 
ment should be seen as normal, 
and leaders in Vermont and Ken- 
tucky should be commended for 
seeking and using independent 
evaluations of their efforts. How- 
ever, impatience and resistance to 
reform often joiti forces to focus 
public attentioti on the itiade- 
quacies of new assessments rather 
than on their potential benefits 
and the progress being made to 
solve their techtiical and logistical 
problems. Policymakers tieed to 
allow' sufficietit time for new 
assessments to be developed, 
technical problems to be addres- 
sed, and legitimacy to be estab- 
lished with educators, parents, 
and the public at-large. 

Costs 

New performance-based tests have 
proved to be more expensive to 
develop, administer and grade 
than conventional multiple-choice 
tests. The demand for increasing 
investments in assessment comes 
at a time when state budgets, and 
local district budgets, are under 
considerable stress. It is hard to 
argue for increasing funds for 
state as.sessment programs when 
state aid to schools is being cut. 
Interstate consortia such as the 
New Standards Project or the 
State Collaborative on Assess- 
ments and Student Standards 
(SC ASS) initiated by the C'ouncil 
of C liiet State School Officers 
have the potential of reducing 
development costs for new 
assessments by spreading them 
across multiple states. However. 




tliey are only now beginning to 
produce products for the stales to 
use, so it is not yet clear if they 
will live up to their promise. 

Instructional Time 

Performance assessment also takes 
more time to administer, so more 
instructional time is lost to as- 
sessment. In the case of port- 
folios, time devoted to revising 
and polishing final pieces can run 
into weeks rather than hours if the 
stakes are high. Increasing time 
for assessment purposes is not 
only costly hut can generate 
resentment among teachers, par- 
ents, and students, aiul increase 
resistance to reforms. 

Public Opinion 

The new performance assessments 
are being opposed by some par- 
ents. Concern about the new 
assessment procedures, or about 
their results, has been expressed 
at one time or another by paients 
in highly regarded suburban 
schools, parents of high school 
students, parents of children 
identified as gifted, and parents 
with lower incomes and levels of 
education. 

It appears that the public has some 
allegiance to multiple-choice tests 
because they are familiar and 
parents feel that they understand 
the results. I’arents also may tear 
that their children will suffer in 
some way (>r lose some competi- 
tive advantage tnmi the imple- 
mentation of new measures ot 
achievement. I’ublic opposition to 
the new asse' .ments in some 
states may also be related to the 
initial small proportions of stu 
dents able to meet higher 
standards. Ibis may be particu 
larly true in suburban commum 
ties long accustomed to hearing 
only good news about the test 
scores in their local schools. 
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Summing Up: 
Improving State Capa- 
city for Assessment 

These are not simple issues. They 
are not just technical issues. They 
will not be .solved overnight. 
Resolving them is confounded by 
debates within, and outside of, the 
educational community. There are 
tho.se who object to standardized 
assessments and believe the only 
acceptable assessment happens in 
the classroom. Such assessments 
may serve valuable instructional 
purposes but are virtually u.seless 
for the purposes of evaluation and 
accountability. Some distrust all 
tests (especially those adminis- 
tered by government agencies), 
and others remain convinced of 
the value of traditional tests. 

There are also those who tear that 
new standards and new assess- 
ments will introduce new biases 
into the system. They want to see 
a new generation of tests char- 
acteri/ed by higher standards and 
better ways of assessing student 
understanding and ability to apply 
knowledge. Others believe that 
state assessment systems can, and 
should lead reform. 

Will We Be Able to 
Evaluate the Impact of 
the SSIs? 

State assessment programs vary 
widely in their ctipacity to pro- 
duce student performance infor 
imition that will be useful for 
evaluating the SSI reform efforts. 
The PSA study team found only 
six state assessment programs in 
mathematics and four in science 
that met till of their criteria. ,A 
handful of other sttites are nou 
developing such assessments. 
They tire shifting tiwtiy from 
minitnutn coinpetetiey sttmdtirds 
tind multiple-choice formats to 
new, more complex assessmetits 




that examine student understand- 
ing of key concepts and their 
ability to apply knowledge. How- 
ever, most of these new assess- 
ment programs will not come 
online quickly enough to con- 
tribute to the evaluation of the 
SSIs. Clearly the potential for 
using student assessment data 
collected by the states to evaluate 
the SSI eitlier as a state or a feder- 
al strategy is limited. Yet the ques- 
tion of whether the significant 
investment in the SSIs has paid off 
in increased student performance 
demands an answ'cr. 

One possible strategy is the use of 
special assessments in those 
schools and classrooms most di- 
rectly involved in SSI activities, 
Ohio and Puerto Rico have elected 
to use this strategy which can be 
implemented faster and at less 
cost than changing an entire state 
assessment program. At least in 
this manner, policymakers can 
determine whether the SSI stra- 
tegy being used in their state has 
the potential for raising student 
performance. 

A second strategy might be for a 
state to customize a commercially 
developed, off-the-shelf mathe- 
matics and/or scietice test and 
administer it in the .Ird, 5th. and 
7th years of the SSI.’' Such a test 
could be administered to schools 
targeted by the SSI as in Ohio or 
Puerto Rico or to a broader sam 
pie. It would be cheaper and 
taster than developing a new state 
test and could provide useful 
interim results. 

However difficult the technical 
and political challenges to broad- 
ening and strengthening state 
assessment systems, the nation 
will not be better off by being 
utiinformed about our children's 
accomplishments atid competence 
iti tiiiithematics atid scietice (or in 
other subject areas). Ignoring the 
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gaps between tlie acliievenienis of 
children in other developed 
nations and our children, between 
boys and girls in our classrooms 
and schools, and among racial and 
ethnic groups, will not reduce 
them or ameliorate their social 
and economic consequences. 

The nation needs good measures 
of student performance in mathe- 
matics and science. In our 
political system, that need can be 
met only by state policymakers 
wh.o are willing to stay the course 
and help put new standard.'; and 
assessments in place. 

Endnotes 

1. Rhode Island is no longer partiei 
paling in the SSI program. I he 2.S car 
rent SSls include PuerU) Rict). reterred 
to as a state for the piirp('ses of this 
report . 



2, This brief summarizes findings re 
ported in Assexsment Programs in the 
Statewide Systemic Initiatives (SSI) 
Slates: Using Student Achievement Da- 
ta to Evaluate the SSI by K. G. Laguar- 
da. j. S. Brcckenridge. and A. M. 
Hightower (Washington. DC: Policy 
Studies .Associates. June 1994). 

.t. American Association for the Ad- 
vancement of Science. Benchmarks fo> 
Science Literacy for All Americans 
(Washington. DC; Author. 1993); Na- 
tional Council of Teachers of Mathe- 
matics. Assessment Standards for 
School Mathematics (Reston. VA; 
Author. 1993); National Council of 
Teachers of Mathematics. Curriculum 
and Evaluation Standards for School 
Mathematics (Reston. VA; Author. 
19S9); National Council of Teachers of 
Mathematics. Professional Standards 
for Teaching Mathematics (Reston. 
VA: Author! 1991); National Science 
Teachers' Association. Scope, Se 
c/Ui-iu e. and Coordination ofSecondars' 
School Science: Volume I. the Content 
Con (Washington. DC: Author. 1993). 



National Research Council. Draft Na- 
tional Science Education Standards 
(W'ashington. DC; Author. 1994). 

4. From SSI National Evaluation field 
notes written by Andrew Zucker. SRI 
International. 

5. The national evaluation is being con- 
ducted under contract with SRI 
International. The Consortium for 
Policy Research in Education (CPRE). 
Policy Studies Associates (PSA), and 
the Council of Chief State School 
Officers (CCSSO) are sub-contractors. 

6. Laguarda. Brcckenridge. and High- 
tower. A.ssessntent Programs. 

7. The Vermont State Board of Educa- 
tion recently extended the mathematics 
portfolio to high schools. It will be 
implemented voluntarily in the 199.‘i-96 
school year but most high schools are 
expected to participate. 

8. Although SSIs have been funded by 

NSF for only five years, many are 
expected to coiitinue work beyond the 
funding period with state and private 
support. ® 
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